Skip to content

Conversation

@jaclync
Copy link
Contributor

@jaclync jaclync commented Nov 29, 2022

Closes: #8235

Description

Since we had an attempted fix for the previous A/A tests where there is a bias toward the control variant, I created another pair of A/A tests (logged-in/logged-out) so that we can see if the bias issue is fixed.

A note on the experiment name: woocommerceios_explat_aa_test_logged_out_202212 and woocommerceios_explat_aa_test_logged_in_202212 experiment names were already taken by @selanthiraiyan's previous PRs that were closed. I re-cloned the experiments with *_v2 names since we can't reuse the same experiment name.

Testing instructions

Prerequisite: a sandbox is required for testing manual assignments on A/B experiments, you can request one if you don't have one - it'd be useful for testing IAP later as well.

There are no visual differences for the A/A tests, you can test it by setting a breakpoint in the logged-out and logged-in state.

Logged-out state

Prerequisites:

  • Please follow the setup in PCYsg-Fq7-p2#assignments-api for proxied sandboxed API requests: connect to sandbox via ssh, and point WP.com API to the sandbox IP address (ifconfig) in the local matchine's /etc/hosts
  • The you need to make an authenticated WPCOM request to manually assign the variant
  • Find the anon ID of the device by setting a breakpoint in ExPlatService on the line urlComponents.queryItems?.append(URLQueryItem.init(name: "anon_id", value: anonId)) and launching the app
  • Make an API request PATCH /wpcom/v2/experiments/0.1.0/assignments with the following body and make sure it's successful:
{
    "variations": {
        "woocommerceios_explat_aa_test_logged_out_202212_v2": "treatment"
    },
    "anon_id": "{{anon_id}}",
    "username_override": ""
}
  • In code, set a breakpoint at a logged-out action callback like StoreCreationCoordinator.start
  • Launch the app
  • Log out of the app or skip login onboarding if needed
  • Tap on Get Started or any other actions that trigger the breakpoint set in the first step. When the breakpoint is hit, check the A/A test variant by po ABTest.aaTestLoggedOut.variation in the debug console --> it should say treatment: nil

Logged-in state

Prerequisites:

  • Make an API request PATCH /wpcom/v2/experiments/0.1.0/assignments with the following body and the bearer token for the wpcom account (I'd use an A8C account in this case), make sure it's successful:
{
    "variations": {
        "woocommerceios_explat_aa_test_logged_in_202212_v2": "treatment"
    }
}
  • In code, set a breakpoint at a logged-in action callback like in HubMenu when an element is tapped
  • Launch the app
  • Log in to a store
  • Go to the Menu tab, and tap on any element in the menu. Or any other actions that trigger the breakpoint set in the first step. When the breakpoint is hit, check the A/A test variant by po ABTest.aaTestLoggedIn.variation in the debug console --> it should say treatment: nil

  • I have considered if this change warrants user-facing release notes and have added them to RELEASE-NOTES.txt if necessary.

@jaclync jaclync added the category: experiments Related to A/B testing and ExPlat. label Nov 29, 2022
@jaclync jaclync added this to the 11.5 milestone Nov 29, 2022
@wpmobilebot
Copy link
Collaborator

You can test the changes from this Pull Request by:
  • Clicking here or scanning the QR code below to access App Center
  • Then installing the build number pr8250-4abc2e8 on your iPhone

If you need access to App Center, please ask a maintainer to add you.

@selanthiraiyan selanthiraiyan self-assigned this Nov 29, 2022
Copy link
Contributor

@selanthiraiyan selanthiraiyan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM :shipit:

@jaclync jaclync enabled auto-merge November 30, 2022 01:11
@jaclync jaclync merged commit 976af81 into trunk Nov 30, 2022
@jaclync jaclync deleted the issue/8235-aa-tests branch November 30, 2022 01:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

category: experiments Related to A/B testing and ExPlat.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Create logged in and logged out context A/A tests

4 participants